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ABSTRACT 
The need for better vaYidation of professional 
licensing and certification exapinatiéns is discussed in Part lI. 
Predictive criterion-related validation studies are difficult to 
develop, time. ccnsuming, impractical and expensive. Since training 
and knowledge variables are ‘not nécessarily sufficient to predict job 
proficiency, it appears logical fo look to performance testing for 
solutions to the problems of validating and revising licensure exams 
more effectively. Four areas which there are differences in 
practice acress differing licensure examinations: (1) the type of 
test, (2) the type of contréls invalved in the testing process, (3) 
the kind of evidence utilized in the entry decision, and (4) the 
range of authcrity and the degree to which the examination provides a 
right to practice are. éfecussed in Part II. Differences in practices 
across these four are appear to be directed toward differential 
contrcl of three primary sources of malpractice--absence of entry 
level prcfessional skills, absence of adequate behavior patterns to 


‘cope with the pressures of professional practice and absence of 


adequate updating of professional practice. (MV) 
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*PART II: 


Analysis of Testing Procedures and Their Implementation 
for Practice, by Jim C. Fortune, Virginia Polytechnic 
Institute and State University, Blacksburg, Virginia 24061 
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As some of you may remember, Jim Fortune and ‘I were en ina 
symposium at the 1974 NCME meeting entitled Validation of Professional 
Licensing and Certification Examinations: A Methodological Dilemma. Jin 
was symposium chairman, and I delivered the overview paper entitled "Over- 
view of Problems Involved in Validating Professional Licensing and Certifica- 
tion Examinations." I plan to use my time as both a review and follow-up of 
this initial paper, again highlighting some problems that I see in this area 
and looking for what changes might have occurred. I will cite examples from 
some of the same professions as discussed in the original paper and review 
what has happened over this two-year span. Jim will follow, citing additional 
professional examples and some additional methodological problems he has 
identified. 

Copies of my originaloverview paper are now available through ERIC 
(ED110491), and since much of the information presented there will probably 
already have been covered by the time we get to this point in the symposiun, 
I will only briefly vouiiaas the points I thought were important both at that 
time and now. 

In 1973, I became involved in the licensure validation issue when I was 
asked by the National League for Nursing to prepare a background paper on the 
validation of the RN (registered sani) licensing examination and related work 
on performance testing. Naively, I thought it would be a simple task of 
consolidating what had been done in other professions. That was a far more 
complex and interesting task than I had expected, and questions and concerns 
raised during the assignment led me to urge Jim to organize a symposium on 
this topic. As far as I know, it was the first such discussion sponsored 


by NCME. I felt, at that time, that the symposium was quite successful, 
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and at least one other person who shared that platform with us, Ben Shimberg, 
is here:today. (Unfortunately, Al Maslow, who was to be a discussant, was 
unable to attend - Al Rabel yhud second chance!) We had a highly attentive 
and interested audience although it was 5:30 before we were done. During 

the lively question-and-anewee period, it appeared that several members of 
the audience were probably far more skilled and versed in this problem area 
than people on the panel. I was gratified by the amount of response the 
symposium generated; it was probably the most successful in which I have par- 
ticipated in ten years ofgattending AERA. 

I hope this symposium and the audience contribution will be equally as 
stimulating. I am particularly pleased to see the issue arising again, two 
years after the first aynpoaiun, and consider it to be a follow-up to what 
we initiated, although the context seems a bit different. 

Let me ‘briefly tell you some of the things discussed in the first paper. 
First of all, I felt it was important to define the difference between 
licensure and certification, as I use those terms. I prefer Jensen's (1972) 
Aigeianebebniiaive he discusses licensure and certification as two types of 
minimum competency testing in that the purpose of the test is to establish 
an individual's status in relation to an established go/no-go criteria. 
Licensing is usually a mandatory program designed to protect the public from 
incompetent practitioners; that is, to prevent an individual with particular 
deficiencies from entering practice. Jensen calls this a "selectirg-out" 
process. Certification, on the other hand, is usually a voluntary program 


where the emphasis is on granting special status to an individual with more 


than run-of-the-mill knowledge, ability, and/or skill. Jensen calls this 


"selecting in." Perhaps the best .known example of a selecting-out exam 
would be for a driving license, where the public is protected from those 
whose knowlege is judged to be below adequate standards. An example of 
selecting-in, or certification, would be the "diplomate" program for medical 
specialities or recent certification programs for automobile mechanics. 
Since validation deals with the purpose to which the test is intended, I 
believe these distinctions to be hadoskans, Unfortunately, the words are 
often used interchangeably. For example, I belive teaching certification to 
be a misnomer, according to this definition, because it is a legal require-. 
ment to begin teaching, to protect the public from incompetent teachers, and 
signifies no special standing within the profession. 

Next, I took a brief look at the sudden growing interest about validation 


of licensing and identified four concerns. First was the criticism of 


testing in general, which in the past decade, has become a popular, head-line 


making cause. Second, there was a proliferation of jobs requiring licensing 


and the hodgepodge of state and local legislative bodies emerging to control 
the process. (Ben Shimberg's 1972 report entitled Occupational Licening 

and Public Policy, is the only comprehensive document I have seen on the 
subject and it provides an excellent overview.) Third, the civil rights 
movement continued to make inroads against discrimination, especially 

here concerned with discriminatory practices in hiring. Fourth, challenges 
were being made to many professions to obtain status through alternative 
training routes rather than the traditional curriculum or school-based methods. 
This is a question of who qualifies to take a licensing exam. In 1972, for 


instance, many returning army medics sought to take the RN exam and were 
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denied such access on the grounds of not baving graduated from nursing school. 
I think it is still safe to say that licensing is experiencing a period of 
attention and questioning for a number of reasons, not the least of which is 
federal legality. Licensing agencies continue to be subject to challenge to 
prove their tests are valid predictors of job performance, measuring job- 


related ‘skills. . ; . 4 


Despite the growing concern about the topic, I found an incredible: lack 


of information, especially research information. This was especiullv true 


in trying to relate licensure to job performance. The information 1 was able 
to locate was scanty, often considered in progress, an’ being done in subject 
matter areas rather than considered collectively as a metho.slogical problem. 
Very seldom was material available through professional ‘ournals and, in some 
cases, professions considered such information confidential. 

Let me share with you something that Maslow, one of our discussants, 
said in 1971 at the Civil Service Commission Research Center, when he was 


advising the Council on Occupational Licensing (p. 339): 


I an convinced that we need to sharpen our ability to 
develop and demonstrate the rattonal relationship between the 
job requirements and the measurement system used to certify 
or qualify people for an occupation. A number of techniques 
are avatlable to tmprove the process of job analysie to get a 
much more exact fix on the critical requtrements for the work 
to be done. I would urge, therefore, that especially in exam- 
tnations for occupational knowledge and proficiency, you insist, 
at the very least, on a clearcut showing of how one proceeds 
from the decision as to the skills and abilities required for 
effective performance to the dectsione that certain or other 
measures will insure that the applicant can adequately perform 
tn that occupation. 


Let's see what changes there have been on some of these points over 


the past two years. First of all, there has been some professional attention 


paid to the matter which I had found extremely lacking previously. Let 
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me cite the APA publication Principles for the Validation and.Use of 
Personnel Selection Procedures (1975) as an example. Issued by the Division 


of Industrial-Organizational Psychology of APA, it is intended to accompany 


the APA publication Standards for Educational and Psychological Tests (1974). 


However, in complimenting this body on issuing the Principles, let me also 


raise a point. This publication says it intends-to provide: (1) principles 
‘upon which personnel research may be based; (2) guidance for practitioners 
“conducting validation; and (3) informacion which may be interpreted for 
personnel managers and others who may be responsible ~ authorizing or 
implementing validation efforts. However, it also says that the Principles 
are not intended to be a technical translation of existing or anticipated 
legislation. I note this publication as evidence of ee professional 
interest; but still there seéms to be a reluctance to relate mthodelocien! 
principles, professional outlook, etc., to the predicaments in which many 
professions now find themselves in relation to va] idating licensing and 
certification exams. We still seem to have methodologists in one corner and 
applied researchers or practitioners in the other. Practitioners are still- 
working in subject fields, and, most methodologists are still daating Gith 
general principles. I have found little evidence that the "Twain has met" 
in the past two years. | 

Let me comment also on the availability of information. It appears to 
me that there is as little overall information available now as there was 
two years ago on how one shoute go about validating a licensing or certifi- 
cation exam, and on methodological issues related to these procedures. 
In two years, I have seen little in the professional journals dealing with 


this issue as a practical problem or even methodologically as a general 
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problem. Hopefully, other people during this symposium will have identified 


work I have been unable to locate. 

Another thing that doesn't seem to have changed is the trend Reward 
the proliferation of licensing exams. I admit that most of these are : 
not what one generally calls professional; for instance, I went to the ; 
Reader's Guide (1973-74) and I found, under licensure, articles with the 
following titles: "Should Auto Mechanics Be Licensed by Law? Pro and Con 
Discussion;" "Licensing Proposal for Sevuiihnad "Debate Over Licensing 
for Consultants;" "How to Strike Back Against Appliance Repair Abuse -- 
Suggestion for Mandatory Licensing;" "Drivers License for Snowmobiles?" 

So the problem of building and validating licensing tests doesn't seem to 
be going away--it seems to be increasing! 

On the professional side, there appears to be some evidence of growing 
. interest in certification exams--exams that grant recognition to a specialty 
area. For example, the American Nurses' Association is now in the process 
of developing a series of certification exams in nursing specialities, such 
as one already available in Psychiatric-Mental Health Nursing (1974). In 
1971, Nattonal Association of Social Workers introduced a written objective 


examination for certification in their field. I'm sure there were numerous 


- 


others. 

_ In terms of legal challenge, I really don"t feel adequately versed 
to deal with this issue, and by this point, hope it has been covered by 
somebody else on. the panel. However, the work of the Equal Employment 
Opportunity Commission, the Civil Service Commission and others involved 


is very important to our topic and needs to be carefully observed and 


interpreted. Unfortunately, information on what is happening with regard 


fe) 


4 


to the legal situation--and especially how that might relate to method- 


: Glogists--is also scarce, 
The Civil Rights Digest, a useful and free publication, devoted its 


Spring, 1975, issue to Job Discrimination and -Affirmative Acti 


on. It includes 
~~ 


an article by White on "Testing and Equal Opportunity" which notes that 
(p. 49): 


bead 


In its recent decision in Moody v. Albemarle Paper Co. 
ithe Supreme Court kept up the momentum of equal employment 


.opportuntty by favoring compensatory back pay awards and by 
spectfying "appropriate standards of proof" in determining 
whether testa_are job related: ee 


The author was actually referring to a 1973 circuit court opinion on 


Moody v. Albemarle which said, with respect to the testing question (p. 139): 


f « 


We think Albemarle has failed in several respects to show 


that ite ’teate are job related, have a mantfest relationship 


to employment, and have been validated tn accordance with EEOC . 
guidelines. .. 


In developing criteria of job performance by which to 
ascertain the validity of its teste, Albemarle failed to engage 
in any job analysis. Instead, test results were compared with 
posstble subjective ratings of supervisors who were given a 
yague standard by which to judge job performance. Other courte 


have expressed skeftioism about ‘the value of such ill-defined 
supervisor appratsals. 


° 


In June, 1975, the Supreme Court upheld and expanded upon circuit 


court ruling that Albemarle had not proved the job relatedness of its 


testing program: This decision gave great weight to the EEOC-Guidelines 


(1974) and also referenced the APA Standards (1974). The court quoted 


the Guidelines concerning use of rankings (p. 25): 
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r The Guidelines provide (that) the work behaviors or. other 
ertteria of employee adequacy which the test is intended to 
predict or identify must be fully described; and, additionally, 2 
in the case of rati echniques. . whatever ortteria are used 

_ they must represent major or critical work behavior as repeated 

- by careful job analysts. 


oa UN 
Let me now address the problem as I see it and as I view it over the 
two year period. First of all, validation studies of licensure exams are 
rare indeed. Seldom is the test development process that sophisticated or 


comprehensive. Often what is offered is some sort of content validation 
‘ Bak ’ ° 


methodology, which varies as widely as calling in a group of professional 
Ce 
cohorts to review questions to doing fairly adequate analysis of critical 
skills. Seldom are such exercises reported except to say they exist: 
However, it would appear to me that predictive eeivertpicealated 

validation studies would be the type most closely fitting the expressed 
‘purpose of ieee exams, that of assuring wives competency on the job 

for the protection of the public. Interest is with the criterion not 

yet obtainable at the time of testing, as one wishes to predict an individual's 
" outcome prior to a situation occurring--that. being when the person is on 

the job. Of Sineaa it can be said that for licensure exams, such valida-’ 

tion studies, the predictive type, are difficult to develop, time consuming, 

impractical for numerous reasons, and expensive. It can also be said that 

psychometric ‘methodology offers little guidance for such studies; the area 

of licénsure lacks classic studies familiar to those ant are schooled in 

psychological testing. Once all this is comprehended, the fact that such 


validations are rare, almost non-existant, is not Siepeterns but nevertheless 


Neeconcerting. 


Research has shown that course grades are not efficient predictors of 


job performance (best demonstrated by the Navy in World War II: Stuit, 


1947; Gulliksen; 1950). We know, for instance, that many written tests 


are found to correlate ‘only too well with reading tests in non-p? ofessiénal 
skill areas, We know ‘there is a lack of relationship between achievement _ 
as measured by paper and pencil tests and performance measures. This has 


been demonstrated in such diverse professions as education (Quirk and others, . 


e ~ 


1972) and engineering (Hemphill, 1963). This —s of correlation — 
great inportance for validation of licensure tests. As far back as 1951, 
Ryans and Fredericksen clearly summed this point up from the measurement 
perspective, (p. 455): .. ; 3 a 


« 
° 


From the standpoint of validity one of the most serious 
errors committed in the field of human meas t has-been 
that which assumes the high correlation of knowledge of facts 
and princtples on the one hand and performance on the other. 
Nevertheless, examinations for admission to the bar, for medical 
practice, for teaching. . . are predominantly verbal teste of 
fact and principle in .the res,ective fields. Relatively little 
attention has been patd to the testing of performance as such. 


If training and knowledge variables are ont necessarily sufficient 
to predict job proficiency, it appears logical to, look next to what is 
broadly called performance testing. © If one accepts the definition of 
performance testing as being a test which is "relatively realistic" 
(Fitzpatrick & Morrison, 1971), then this is a logical place to look for 
the answer to: (1) how to validate licensure exams more effectively, 
and (2) nol to revise licensure exams themselves if necessary. 

The’most interesting and well documented use I found concerning 
performance measures in predictive research was in the area of empléyee : 


« 
selection and promotion. Of course, the employer situation has numerous 
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advantages over licensure boards, such as: control over subjects, limited 


range of jobs, job descriptive information, length of period of observation, 

and the-possibility of gradually implementing a testing program, giieaine 

vesennch time to study predictions before changing the incteion process. 

In this field, assessment centers are an effective performance-based type re ~ 


of employment or promotion screening device. The procedures stimulate 


situations that would be faced if the examinee were to be moved up to a 
higher grade position, and using a sophisticated form of role playing, | 
develops information regarding how he might cope with decison making. 


Validation studies done on assessment center techniques have shown them to 


be a better indicator of future Success than any other’ tool management has 
yet devised. (Bray and Campbé1l, 1968, describe how one such center works 


and includes the validation.process.) Although ‘the assessment center concept 


‘could be used as a'validation tool for licensing exams, as an on-going 
‘ technique, where large numbers of people must be cesred each year, it is 
obviously unrealistic. | { 

Illustrative of a mee practical approach to introducing performance 
into testing situations are two types of programmed testing developed in 
the medical ‘profession. In the assessment of clinical competence, perform- 
ance is simulated on objectively scored paper and pencil tests. The 


National Board of Medical Examiners first introduced the concevt (Hubbard, / 


1964) and now use programmed testing for the médical licensing exam Rart 
III on clinical competence, which previously was ‘a practical bed-side 
type of oral examination. There are two competing models in the medical > 


profession for this type of testing. In both models, the examinee is con-_ 


fronted by a realistic clinical situation and proceeds through a series of 
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decision choices, each step accompanied by an increment of information upon 
which the next depends, similar to programmed teaching. The linear model 
used by the National Board allows’ for only one correct option at each 
decision point while the branching model promoted by McGuire and Babbott 
(1967) allows for more than one route to a solution and is used in specialty 


° 


certification. 
To my knowledge, neither variation has been validated in relation to 

predicting job performance. The Part III, or clinical competency exan, 

is said to derive its validity by measuring something different from what 


Parts I and II measure, which is strongly related to medical school course 


work; scores on Part III correlate only moderately with those’ on Parts I 


and II. Cronbach (1970, p. 444), having reviewed this so-called validity 
evidence on Part III notes: "Follow-up studies are needed to make sure ~ 
the test measures a skill of medical practice and not just ingenuity in 
test taking.” 

Similar to problems confronted by those attempting predictive validation 
of licensure tests, performance tests development logically begins with an 
identification of, specific skills and abilities involved in the a ee 
the test is designed to predict. The next step is the choice of representa- 
tive tasks, a difficult task which strongly influences the validity of 
the performance tests. Other difficulties with performance testing come 
from a lack of applied methodology in that performance tests are by nature - 
criterion-referenced and procedures for estimating reliability and validity 


are still subject to debate. 


I would like to digress on this last point, a problem area which I 


has been seriously neglected by methodologists. As I noted previously, 


licensure and certification exams are types of minimal competency exams 


. and like the performance measures we have been discussing, would normally 


be considered criterion-referenced. The examinee is theoretically tested 
in terms of absolute criteria; how well the person can do what he or she 
needs to do to perform adequately ee the job. Comparison among test takers 
is not the purpose of licensing exams. However, most licensing exams on 
which I have reviewed research data are still developed on norm-referenced 
models. 

Returning to performance fests, most of the literature discusses them 
as a new form of dium: used to increase the realism of the test. My 
primary interest in performance tests is less commonly discussed, that of 
providing criteria for predictive valjdation. The only suggestion of such 


research I have located was a theoretical discussion on "Providing a 


Criterion Measure" also by Ryans and Frederiksen (1951, p- 466): 


Performance test data may provide, first of all, a crtterion 
for research. Information ytelded by performance teste makes 
, posetble the validation of other measures which, although of a 


& more tndtrect nature, may be more economical tn administration. 


In many sttuations, tt ts difficult and expenetve to admintster 
performance tests to large nuwnbers of examinees. Such situations 
demand the construction of psychometric instruments that will 
yteld measurements related to criterion and will be also practt- 
cable. In the construction of aptitude teste for various skills 
and operations, performance teste may provide the criterion 
against which the avatlable second order test can be judged. 


Looking to specific professions and validation efforts, I would first 
like to thank the National Board of Medical Examiners, Educational Testing 
Service, and the National League of Nursing for supplying me with the 


information for this section. The information was obtained through personal 


contacts and most is not generally available. 


—_ 
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The National Board of Medical Examiners appears to have no new vali- 
dation studies available on the Hattonal Board Examinations Parts I, II, and 
III for medical students. (THe two studies sent to me were both over ten 
years old.) However, in 1974, the National Board formed a R & D Department, 
and their correspondence says several developmental studies which include 
validation components are in progress. 

Two validation studies of certifying exams in the medical profession I 
can report on concern practitioners of internal medicine and physician 
assistants. Schumacher (1973) reports a valuation study of the written 
examination for internal medicine which was based upon comparing test perfor- 
— of first and third year residents. According to the author, the. study 


attempted to (p. 132) ". . . determine whether the examination is sensitive 


to gains in knowledge, ability, and skill that presumably oecur during graduate 


training in internal medicine." The study was said to (p. 133) ". . . support 
the hypothesis that the examination had validity for measuring achievement 
at the graduate level." The study makes no pretense of relating achievement 
to expected future performance or even a sound job analysis. 

The Physician Assistant Certifying Exam, new in 1973, is also said to 
have validity in that those with formal eataabing scored higher than those 
just beginning training. However, the report (National Board of Medical 
Eeanitiave, 1973-74) also noted evidence of construct validity in that post- 
graduates with clinical experience after graduation did better on the exam 
than those without, though experience prior to entering training had not 
proven significant. An additional study to estimate concurrent validity 
compared program Qe rating of students on 40 statements of clinical 


competence with examfnating. Four of the six examination components cor- 


related significantly (though the highest was only .20) with at least one 
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of three rating factors. The report indicates modifications will be made 
in the following year to refine the rating scale. However, the report does 
not deal with predictive validity nor is there any mention of content or 
job analysis im developing the rating form. 

The National League for Nursing? is proceeding with plans to validate 
the RN licensure exam, and in designing the project, has accepted ". .. as 
a first principle, [that] the sampling of content for a new test should 
be altered in such a way that items drawn from on-the-job experience would 
receive precedence over items based mainly on knowledge." The rationale 
- given is that licensure is meant to assure safe and effective practice, and 
to evaluate nursing practice would require judgments as to whether the 
decisions and actions taken in sample cases is appropriate. In shifting 
from curriculum-based learning to job-related behavior, the question becomes 
how to sample items from the job situation. The League with A.I.R. (American 
Institutes for Research) is currently carrying out a critical incident 
study which will testes the empirical base for a criterion-referenced test. 
Almost 14,000 incidents were collected in five specialty areas with geo- 
graphic and institutional diversity. Classification of incidents and test 
specifications are the activities of the current phase. 

The League then plans to construct a performance simulation test and 
compare results with the current RN exam. It is currently seeking funding 
to develop more elaborate performance simulation exams and to attempt vali- 
dation by observation of performance. (This appears to be by the most 


active professional group on the validation scene to date!) 


1onis information is drawn from a draft proposal which the author has per- 
mission to use but which is not generally available and thus no reference 
is noted. (See Hecht, 1974, for additional information on the RN licensure 


exam and related studies.) 


16 


Looking to some professional éxams that were not reviewed at the last 
session, Educational Testing Service has been involved in certification or 
licensure test development for social workers, lawyers, and nursing special- 
ists. 

in 1969, the National Assoctation of Social Workers required the 
addition of an objective written test to the membership requirements of 
the Academy of Certified Social Workers. Since 1973, ETS has been adminis- 
tering that test. Research on the test reported by Boyd (1975) noted that 
no item clusters were large enough or differentiated enough to provide sub- 
scores--subjects doing well in one subject area would do well in another. 
Biographical data showed Blacks with consistently higher failure rates. 
Further study (Sharon, 1975) recommended reduction of jargon and wordiness, 
that study guides be senvided, and that the test be empirically validated. 

A validation study for the social worker certification exam has just been 
approved. The information I was sent. states the exam must grant certification 


to competent workers regardless of test-taking ability, and deny certification 


to incompetent workers regardless of their theoretical knowledge. (Sounds 


more like a delayed licensure exam!) The validation method outlined uti- 
lizes supervisor rankings on overall competence and on several specific 
dimensions of job pecformance related to a restructured exam. If pairs of 
raters agree, the criterion is assumed to be reliable and items thus developed 
to relate to effective job performance. No mention of job analysis is made. 
The plan also states, among utilization of study results, that such infor- 
mation will provide evidence regarding fairness in case of litigation. 

A validity study on the multi-state bar exam is near completion (draft, 
Carlson and Werts, 1975) and will be reported on at both NCME and AERA 
meetings by Dr. Al Carlson. The purpose of the study was to investigate 


ee 
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the relationship among the Multi-state Bar, bar exam essays, law school 
grades, law school admission tests and undergraduate scores. The summary 


Suggests reasonable relationships were found among these measures. Though 


this study does describe the relationships of the exam to previous achieve- 
ment/aptitude measures of law students, it does not appear to be concerned 


Therefore, the exam's validity 


with predicting adequate job performance. 
as a licensing instrument appears in need of further study. 
No research information was made available to me on the Psychiatric- 
Mental Health Nursing Certification Examination sponsored by the American 
Nurses' Association and developed by ETS. 
Given the research information previously cited in\ this and the pre- 


vious paper, it would appear that most professions have a long way to go in 


adequately validating licensure and certification exams. Few studies are 
directed at predictive validity or the performance which can be expected 
on the job. Many seem content to anlar exams to academic measured based on 
curriculum content and teatniing which have no proven relationship with 
the tasks to be performed. Most exams are put into use before ealdaey 
studies are started or with none in mind until problems identify themselves. 
And those professions I have cited here are probably further along than 
most. But given recent legal decisions, time seems .to be running out. 
An additional challenge to professional individual licensure is 
institutional licensure. Interestingly, it appears most vocal in the 
one profession which has appeared to take leadership in licensure valida- 
tion--nursing and other health care occupations. Agree (1973) reviews the 
movement in health care and notes support from the Department of Health, 


Education, and Welfare in testing various plans of institutional licensure 
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for health personnel. Both the A.M.A. and the N.L.N. have formally opposedey -f 


institutional licensure in lieu of individual licensure for doctors and 

nurses. If nothing else, this alternative should further prompt action 

toward upgrading licensing and certification programs and can be viewed 

as an added critical pressure on professions to police themselves wisely 

and equitably. NY 

In closing, I think the field of licensing and certification continues 
to present methodologists with real and immediate challenges. Here are 
practical problems based on real and current namie: If each occupation 
continues struggling on its own, without serious attempts from a group 
(such as we have here today) to. provide integrated conceptual and method- 
ological frameworks, solutions will remain a long way off. 

In two years, since I last wrote that closing statement, I. have seen 
little to indicate a cohesive eabindoioeieal effort being made to tackle 
any of the problems mentioned herein. Professional licensing and certifi- 
cation exams continue to be developed and used without adequate validation 
data. What work that is being done tends to be the dilemma faced by those 
responsible for licensure and certification. Few share their experiences 
or the knowledge gained. Hopefully, panels like this will create more 
interest in this area of applied methudology. If there is another panel 


in two years, we hope to see significant progress being made. 
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In the second part of our paper, we present ‘an analysis of the licen- 
sing certification processes as they are practiced by the different profes- 


sions. This analysis includes an investigation of different procedures 
. ZL 


used in the development and the administration of the licensing examination. 
The investigation will focus upon the roles and benefits of particular 
practites and will attempt to determine their implications for the educational 
professions and for competency-based education. 


An Analysis of Licensing and/or Cértification Examination Processes as 
Practiced by Different Professions 


I.would like to look at the variety of practices which exist in four 


e 


different areas of the licensing examination process. These four areas 


are: (1) the type of test; (2) the type of controls involved in the ‘. 


4 


testing process; (3) the kinds of evidence utilized in the entry decision; 
and (4) the range of: authority and the role of the examination as it re- 


4 


_ lates to right-to-practice. oe 


Type of Test. Four factors should be considered in describing the 


s 


differences in the type of test or licensing examination used by different 


professions. These four factors intlude: (1) the construction of the test; 


(2) the measurement strategy used in the test; (3) the type. of setting in 


which the test Is given; and (4) the scoring process. With regard to the 
construction of the test, several practices are used by ee professions. 
In some sevtenstionn, the licensing or certification examination is made up 
by trainers in that profession. « Such examinations usually reflect the 
training curriculua wuich has been required for entry into the given pro- 


tession. The conte it of other examinations is provided by practitioners 
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already in the profession. This kind of examination iecdowigned Eo peat 

on content which comes from practice of the profession and which is 

deemed necessary for successful practice in that profession. A seta option 
in the construction of professional licensing examinations iis the develop- 


~S” 
ment of the test through contract with a professional test development firm 


or psychometrician. The developers of the test usuafly use content extracted 
from the training curriculum and from work samples taken from the field of 
practice. Still a fourth tulle in the censtruction of the test is to take 
items from either a theoretically-based or empirically-generated set of pro- 
fessional competencies. ie stan cases, these competencies have come from 
panels of experts--both practitioners and trainers--who were asked to identify 
competencies through some systematic procedure of brainstorming. In other 


é ’ 
cases, competencies are derived from task analyses of professional records 


’ 


or behaviors. 
Each of these four processes of .test development maintains a certain \ 
degree of face validity relative to the measurment of skills considered 
by an identified reference group to be essential for prafessional practice. 
These different routes tend to generate differences in content but appear , ‘ 
to offer equal opportunity to utilize appropriate test construction tech- 
niques. The various examinations, however, do not. appear to have undergone 
the same level of test analysis or to have been d@veloped.at the same 
level of psychometric sophistication. Present legal issues indicate a need 
to gather more evidence on the validity of the examinations. for individual 
Assessment uses. i 
The second consideration in regard to type of test is the eer 
stratey espoused bv the examination. Most of. ihe licensing examinations 


used by the professions are fixed-response, machine-scorable, norm-referenced 
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Poe, : a é \ 
. examinations. This form of examination has been adopted primarily for 
convenience. Also, there is a need in licensing examinations to main- 
tain security of the test. ‘The most convenient way to nixiniee security a 4 
is to use a random, evolving-item pool; and aft Wiictiaaie items are the 
easiest to construct for item-pool abidanttens Gene, in licensing 


examinations, there is a need to process a large number of tests rapidly 
and secuienn ey, Sneedvant accuracy are assured by: machine scoring. The , oe 
Keenereaital characteristics of these enue prolabty came into being 
because the infancy of criterion-referenced measurenent and logistical 
difficulties of' performance-based tests limited the use of these aT an 
‘Tt ds ‘anticipated that test strategy options based upon penfoxsance and 
criter#on-referenced testing will become practices of the future since ' 
the easurement “assumptions in such strategies ave so appropriate to iicen- ° 
sing examinations. Given that the primary teabise cixuanite to guarantee 
‘ the presence of given competencies, the criterion-referenced strategy of 
measurement is an extremely promising means to reform licensing examination 
‘ procedures. Furthermore, it appears that increased evidneces of the 
predictive relationship between the licensing newckvadtion and competency 
- on 

‘in practice are essential ‘to. guarantee the civil rights of, all. Performance 
testing offers one way to iuiescanan tile predictive relationship. 

The third factor to be considered in describing variations in type 
of test ig the setting in whatch the test is administered. In some examina- 
tions col where ‘individual Getingestnn or simulated practice is required, 
the test’ is individually administered. Most often, however, the licensing , 
examination is given in a public setting at a time of year selected for 


sliert convenience (-.g., ihe exam may be yiven immedtately at the end 


: of 4 triining provrim or at the beginning of a seasonal practice period). © 
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This ‘kind of testing process involves travel of the examinee to the test 


site and the administration of the examination materials cuted weandudioend 
conditions by a professional examiner. Other examination settings might 
Seine hist Recatagn of & danticulas training program or an employement 
center. The ‘primary effacts of the setting on the examination procedure 
‘appear. to be related to the degree to which the examination environment * - 


is removed from the field of practice and to the degree to which the setting 


creates anxiety. , ; © 


. +? - rs 
’ 


The fourth consideration to be made in terms of the type of test is . 
related’ to’ the grading or storing procedure. Some espe of items require 


more subjective judgment for grading than do other items which are e the 


fixed response, factual content nature. Often, the items that require 


> x 
subjective judgment in their scoring can be hypothesized to contain greater * 
content validity with regard to actual professional practice than do factual: 


items. In the scoring of professional licensing examinations, practices 


a tp range from the subjective judgment of peers, whose. objectives are to 


. 


protect their profession from entry by ABCOmPEGARES, ‘to rote marking of 
mastery~type items dealing with skills or ihacniliadigas deemed essential {oe 
professional praceive, Although the subjectivity-objectivity continuum : sian 
of scoring is not Sicnsiy ialactabes with who grades the test, test 

scoring relative to who determines what is the right answer is another 

area in which society avis by peatauston. As mentioned, the correct— 

ness of answers is sometimes determined by trainers, sometimes by practi- 

tloners in the field, and sonet ings by mixed committees. ser aa the 

breides foster entry into the profession; practitioners end to policy 


- 


entry ‘nore carefully.’ ‘ 26 , 


a ra 


The Type of Controls Involved in the Testing Process. A second area 

of practice that is worthy of concern in professional licensing procedures 
is contr-~ over the examination process. The first consideration in look- 
ing at controls is the agency or type of group which is responsible for 
waking the entry decision. Four kinds >f groups typically police professions. 
Generally, the profession polices itself through an association. This 
association, such as the American Medical Association, frequently maintains 
control nationally over the licensing examination. In other cases, such 
as law, professions police themselves through state associations. Bar 
examinations, with the exception of subscribers to the multiple state bar 
examirutions currently being offered *y ETS, provide control of entry 
at the state level by colleagues in the law profession. Any a¥vancaga 
gained through local control may be sacrificed by the-requtrementa of 
reciprocity and bv the necessity to tolerate nonuniform standards of entry 
into "he profession. A third tvoe of -onttol Lavolved in the testing process 
te direct control by the state government, as is exercised by the certifica- 
tion of teachers. Although a professional association is indirectly involved 
in i.e certafication process by recommending standards, certification re- @ 
quirements for teaching are established primarily by each state department 
of education. Often this certification right is delegated by the state 
department to training institutions who certify on the basis of successful 
completion of courses, rather thin successful performance on an examination. 
In suze circumstances, training institucions gequtine a comprehensive exam- 
‘ aba ¢ ,otential graduates over a required curriculum. A fourth 
source of testing ccntrol ts that which i» available to a potential employer 

af eatte f exasination vider 2 gained at a local level. _ 


. t examin tf is often given by enployer's personnel offices and 


Pa wat ot hatrery - phy a testing firm. 
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Two additional means of control have at times been practiced by pro- 
fessions. These two means of control, although used less often than the 
controls described above, do provide options which may, in some cases, 
be useful to the profession. The first of these options is control or 
certification through a subcommittee of a professional association. This 
subcommittee fulfills the certification or accreditation role with support 
from the profession as represented by association membership. The National Fe 
Council for the Accreditation of Teacher Education provides a good example 

: of this practice. The final element of control comes through a govern- ° 
- mental agency in the form os public protection law. Although not direct- 
ly involved with a profession, licensing procedures for oilote utilize s 
this type of professional entry control. 
: ; The ee control of entry into a profession seems to lie either 
in the profession itself or in a subdivision of government. It would 
appear that both state government and professional association controls 
appear useful and essential in cases where public safety is an issue and 
where clientele may not have the choice of selecting the practitioners. 
Reetonal or state control usually results in non-uniform standards, which 
require the practitioners to undergo reexamination or to otherwtse qualify 


for reciprocity before they can move their practice across a regional or 


Ss - state boundary. Hence, the uniformity of standards generated by national 
= licensure control provides a mechanism through which professionals can 


avoid (ue nuisance of reexamination and/or states ‘can avoid the risks 


involv ! with rec (*coctity. Most often, however, national licensure tends 


' e 


to occur only at the entry point in a profession, which fails to assure 


28 


up-date capability to keep practitioners in the profession aware of current 
developments fs. i profession. 

A second consideration related to control over the testing procedures 
involves options available at the application level for professional exam- 8 
ination. These options vary from required membership and petition for 
examination through a professional association to application and fee 
registration. Other options involve testing at the close of a training 
program or testing at the time of application for employment. In cases 
where fee application is required, more emphasis is usually placed upon 
the test results and there is more standard application of the test content 
to the entry decisiun. When the examinee must belong to a professional 
organization and must petition that organization for the right to be examined 
for entry into the profession, then it is more likely that prior qualifica-— 
tions and training become additional considerations for entry into the 
profession. 1n some cases, more weight is placed upon qualifications than 


upon the results of the examination. 


lhe Kinds of Evidences Utilized in the Entry Decision. A third con- ‘ 
dition that should be examined in ascertaining right of entry into a pro- 
fession is the type of supplementary evidence that is used in either 
dererataing eligibility for examination, or for determining professional 
entry. Traditionally, training, experience, and course work have been 
major considerations in ascertaining a candidate's qualifications. In the 
ccs: °f law, completion of a trining program is required before the bar 
examination can be taken. Tn some cases, including teaching, the completion 


of sneeiffed coir cor with « suininel grade point qualification is all 
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that is needed to enter the profession. During the lat ‘ter phases of the 
educational training program, potential teachers go through a period of 
ae ‘ 

practice teaching, which is usually one of the requirements for certifica- 
tion. This experience varies in the degree to which the practice experience 
approaches a longitudinal entry into the profession. -In some cases, such 
as medical licensing, entry is possible only after the completion of both 
a training program and a term of supervised professional nantes ("residence"). 
The residence in medicine more closely approximates professional practice. 
It can be hypothesized to provide a better gauge of how the potential. 

, practitioner will perform his or her career endeavor. * 

An additional source of evidence which may be examined prior to a 
candidate's entry into a,profession is peer acceptance,- which is usually 
gained through the solicited sponsorship of a candidate by a member already 
in the profession. The American Psychological Association, for example, 
requires sponsorship by two members prior to a candidate's admittance to 


that association. A-candidate usually earns peer acceptance through his 


association with members of a profession during this training or apprentice- 
ship. 


The Range of Authority and the Role of the Examination as it Relates 


to Right-to-Practicé. It appears that in licensing examinations the range 


; | 
of authority varies from simple certification or accreditation by the pro- 


fession to actual]. blockage of practice in the field. For instance, non- 

\ 
satis i teachers may teach with only a minor penalty in terms of 
compensation for services rendered. Licensing or certification of -psycho- 


lovzical testers is based upon a graduated scale. Levels of examiner 
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sophistication are determined, and less qualified persons are barred from .~ 


¢ 


using some types of tests. In the medical profession, licensing is essential 
to any practice at a‘national level. Bar association tests tend to keep 
underqualified persons from the full-fledged practice of law. 

~ Te appears, therefore, that two types of power or authority can be 
associated with professional examinations. The first is associated pri- 
marily with the licensing examination, and is that of barring the unquali- 
fied from practice. The examination tends to watanittak the existence 
of a minimum level of skill _and knowledge, which has been deemed essential 


for successful professional practice. The.examination is intended to 
police the profession of unqualified people, and also to raise the proba- 
bidaty of capable professional practice. Often, however, the examination 
ascertains evidence of minimal qualifications only and fails to produce 
a evidence of future success in practice. 2 

A second power or authority assigned to certification examinations is 
the power to identify expertise in a speciality area. Such examinations 
are designed to provide evidences of special skills or knowledge that. 
are currently held by practitioners in the profession. These examinations 
must be designed to demonstrate the mastery of special competencies or 
custetendiing capabilities. In order to produce evidences of individual 


advancement, the examinations must possess the capability to discriminate 


agen those professionals who have that special capability and those who 


do not. 
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Components of a Licensing Examination Process which are Functional in 
Licensing and Certification for the Professions 

In order to identify what is required of a profession's licensing or 
certification process, it may be helpful to investigate the causes of 
malpractice by the professionals. Failure to provide adequate professional 


services can be traced to three principal causes. First, some professionals , 
4 . 

fail to stay current “in their field, and the level of practice in their _ 
prdfessions advances beyond their own level of expertise. ‘Second, some 
professionals may enter the profession with an adequate set of functional 
competencies but may fail ne make personal day-to-day adjustments to prac- 
tice. In such a case, professionals may develop a personal problem or 
habit which interferes with professional practice and care for the client. 
And third, some professionals fail in practice because they lack adequate 
competence in one or more of -the fundamental skill areas needed for success- 
ful practice in the profession. It is my opinion that this last reason 
for failure is reduced to a minimum by current examination stacticée. 

It is believed that malpractice in our professions today stems pri- 
marily from one of the first two causes of failure. ‘If this is true, 
what implications does this have for future practices in professional 
certification and licensing? What are the implications for the recent 


movement toward competency-based education and competency-based professional 
training programs? I think it becomes quite apparent that a periodic 
review or reexamination of professional practitioners can reduce the inci- 


dence of failure due to the first cause, the inability of the practitioner - 


to stay current. Such rapidly developing fields as dentistry require 
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the practitioner to invest time and effort in keeping abreast with the 
field. Licensing at the close of a training program will not insure that 
professional growth and updating of practices will occur. Hence, it becomes 
apparent that reexamination of.licensed professionals should occur with a 
frequency commensurate with the speed at which the profession is developing. 

The second primary cause of failure in the professions, namely, the 
lack of adjustment to professional practice and the development of personal 
habits incompatible with good professional practice, cannot adequately be 
dealt with within the context of the licensing aestatantens setting. Trike, 
this problem must be dealt with in the setting of professional practice. 
Here, the medical profession's model of using a period of residence seems 
to offer a potential solution. To translate this practice into the area 
of teaching, one might suggest that a teacher teach for three years under 
the supervision of an established colleague prior to his being given full 
professional certification. To some degree, it can be argued that the prac- 
ticé teaching experience satisfies this level of practice. ‘However, the 
degree to which the practice teaching experience fails to simulate the 
actual teaching situation is directly associated with the degree to which 

' 

practice teaching fails to produce the longitudinal personal evidence needed 
to predict success in Practice, death) requirements do provide a greater 
amount of longitudinal evidence ean can be gained in oa txietoatiton setting. 
Threats of disbarrinent eeeaeetihd dite legal profession can extend the guar- 
antee of adjustment over a Inger period ‘of professional practice. 


It appears that criterion-referenced testing procedures are alsd 


, ir Se 
essential to complete the revision of the testing-out process... The. cri- 


terion-referenced testing prockduned-alloufor.@ ten eaenmabdditnheten 
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of competencies and for a direct association bf test content in those 
competencies. Current practices in licensing fail to generate all of the 
e 
validation evidence necessary to make accurate and unbiased assessments 
of individual eonpecunee and potential. The criterion-referenced testing 
strategy can, with proper validation, measure competencies which can be 
“defended as fundamental to professional practice. It could be argued 
that the use of norm-referenced tests may be used primarily in the cheek 
fication of exemplary proficiency and may in fact satisfy one aspect of 
the testing-in function of certification examinations. Such would not 
be the case, however, in the identification of mastery of specialized 
skills. Expertise can be defined in many ways; exceptional practice can 
be defined either in terms of relative proficiency in skills beyond minimal 
expectations, which is a norm-referenced testing task, or as mastery of 
specialized skills, which is a criterion-referenced or mastery-testing 


task. 
Summa r 


In ection Licensure examinations differ significantly in six major 
, areas, namely, (1) type of test; (2) test content; (3) administration; 
(4) scope of authority; (5) purpose; and (6) point of effectiveness. 
Differences in the type of test relate to testing strategy; ie., whether 
it is norm-re*érenced, criterios-referenced, or performance-based. Usually 
staff and time limitations force tic test into an objective test mode which 
is machine scorable. The content of ti licensing examination may bé taken 
1 


from the training program, from the observation of practitioners at work “ 
v 


or from mixed content selected by professional test makers. Usually 
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examinations built on training content are made up by members of the pro- 
fession who have gone to some form of professional Keene: Test con- 
tent coming from the field of practice usually contains items submitted by 
selected practitioners. Modes of test administration differ in that some 
tests are given by professional organizations at the end of a training 
-program; come are commercially subscribed either at the point of application 
for a job or nationally through government agencies. ,The range of authority 
fora licensing examination may extend over state, nation, professional 
association, including either license to peaniten, accreditation, or special 
recognition. The purpose of the rest may be to test people out for lack 

+ of competence or to test in to note special qualifications. Differences in 

the point of effectiveness range over the entry period or extended period 
of practice and the ability to be updated. , 

7 All of these six areas of difference must he considered in the design 
of a licensing examination for a given purpose and in rhe. gutlwedag of 
evidences of the quality of a given licensing exam. An analysis of today's 
practices in medicine, law, and education suggests that these professions 


have failed to adequately consider all of these six differences in the 


design of their current licensing examinations. 


oe 


